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(57) Abstract: A method and apparatus for automated cell analysis of 
biological specimens automatically scans at a low magnification to ac- 
quire images (288) which are analyzed to determine candidate cell ob- 
jects of interest. The low magnification images are converted from a 
first color space to a second color space (290). The color space con- 
verted image is then low pass filtered (292) and compared to a threshold 
(294) to remove artifacts and background objects from the candidate 
objects of interest pixels of the color converted image. The candidate 
object of interest pixels are morphologically processed (296) to group 
candidate object of interest pixels together into groups which are com- 
pared to blob parameters (298) to identify candidate objects of interest 
which correspond to cells or other structures relevant to medical diagno- 
sis of the biological specimen. The location coordinates of the objects 
of interest are stored and additional images of the candidate ceil objects 
are acquired at high magnification. The high magnification images are 
analyzed in Ihe same manner as the low magnification images to con- 
firm the candidate objects of interest which are objects of interest. A 
high magnification image of each confirmed object of interest is stored 
for later review and evaluation by a pathologist. 
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METHOD AND APPARATUS FOR AUTOMATED 
IMAGE ANALYSIS OF BIOLOGICAL SPECIMENS 

CLAIM OF PRIORITY 

5 This application is a continuation in part of U.S. 

Patent Application Serial No. 09/344,3 08, filed June 
24,1999, which claims priority to Provisional Application 
Serial No. 60/129,384, filed April 3, 1999, and is a 
continuation of U.S. Patent Application Serial No. 
10 08/827,268, filed 
March 28, 1997. 

TECHNICAL FIELD 

The invention relates generally to light microscopy 
is and, more particularly, to automated light microscopic 
methods and an apparatus for detection of objects in a 
sample . 

BACKGROUND OF THE INVENTION 

In the field of medical diagnostics and research 
including oncology, the detection, identification, 
quantification and characterization of cells of interest, 
such as cancer cells, through testing of biological 
specimens is an important aspect of diagnosis and 
research. Typically, a biological specimen such as bone 
marrow, lymph nodes, peripheral blood, cerebrospinal 
fluid, urine, effusions, fine needle aspirates, 
peripheral blood scrapings or other materials are 
prepared by staining the specimen to identify cells of 
interest. One method of cell specimen preparation is to 
react a specimen with a specific probe which can be a 
monoclonal antibody, a polyclonal antiserum, or a nucleic 
acid which is reactive with a component of the cells of 
interest, such as tumor cells. The reaction may be 
detected using an enzymatic reaction, such as alkaline 
phosphatase or glucose oxidase or peroxidase to convert a 
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soluble colorless substrate to a colored insoluble 
precipitate, or by directly conjugating a dye or a 
fluorescent molecule to the probe. Examination of 
biological specimens in the past has been performed 
5 manually by either a lab technician or a pathologist. In 
the manual method, a slide prepared with a biological 
specimen is viewed at a low magnification under a 
microscope to visually locate candidate cells or objects 
of interest. Those areas of the slide where cells of 

10 interest are located are then viewed at a higher 

magnification to confirm the objects or cells, such as 
tumor or cancer cells. The manual method is time 
consuming and prone to error including missing areas of 
the slide. Automated cell analysis systems have been 

is developed to improve the speed and accuracy of the 

testing process. One known interactive system includes a 
single high power microscope objective for scanning a 
rack of slides, portions of which have been previously 
identified for assay by an operator. In that system, the 

20 operator first scans each slide at a low magnification 
similar to the manual method and notes the points of 
interest on the slide for later analysis. The operator 
then stores the address of the noted location and the 
associated function in a data file. 

25 Once the points of interest have been located and stored 
by the operator, the slide is then positioned in an 
automated analysis apparatus which acquires images of the 
slide at the marked points and performs an image 
analysis . 
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SUMMARY OP THE INVENTION 

A problem with the foregoing automated system is the 
continued need for operator input to initially locate 
cell objects for analysis. Such continued dependence on 
5 manual input can lead to errors including cells or 
objects of interest being missed. Such errors can be 
critical especially in assays for so-called rare events, 
e.g., finding one tumor cell in a cell population of one 
million normal cells. Additionally, manual methods can be 

10 extremely time consuming and can require a high degree of 
training to identify and/or quantify cells. This is not 
only true for tumor cell identification and detection, 
but also for other applications ranging from neutrophil 
alkaline phosphatase assays, reticulocyte counting and 

15 maturation assessment, and others. The associated manual 
labor leads to a high cost for these procedures in 
addition to the potential errors that can arise from 
long, tedious manual examinations. A need exists, 
therefore, for an improved automated cell analysis system 

20 which can quickly and accurately scan large amounts of 
biological material on a slide. Accordingly, the present 
invention provides a method and apparatus for automated 
cell analysis which eliminates the need for operator 
input to locate cell objects for analysis. 

25 In accordance with the present invention, a slide 

prepared with a biological specimen and reagent is placed 
in a slide carrier which preferably holds four slides. 
The slide carriers are loaded into an input hopper of the 
automated system. The operator may then enter data 

30 identifying the size, shape and location of a scan area 
on each slide, or, preferably, the system automatically 
locates a scan area for each slide during slide 
processing. An operator then activates the system for 
slide processing. Alternatively, the processing 

35 parameters of the slide may be identified by a bar code 
present on the slide or slide carrier. At system 
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activation, a slide carrier is positioned on an X — Y 
stage, the entire slide is rapidly scanned at a low 
magnification, typically lOx. At each location of the 
scan, a low magnification image is acquired and processed 
5 to detect candidate objects of interest. Preferably, 
color, size and shape are used to identify objects of 
interest. The location of each candidate object of 
interest is stored. 

At the completion of the low level scan for each 
10 slide in the carrier on the stage, the optical system is 
adjusted to a high magnification such as 40x or 60x, and 
the X-Y stage is positioned to the stored locations for 
the candidate objects of interest on each slide in the 
carrier. 

is A high magnification image is acquired for each candidate 
object of interest and a series of image processing steps 
are performed to confirm the analysis which was performed 
at low magnification. A high magnification image is 
stored for each confirmed object of interest. 

20 These images are then available for retrieval by a 

pathologist or cytotechnologist to review for final 
diagnostic evaluation. Having stored the location of each 
object of interest, a mosaic comprised of the candidate 
objects of interest for a slide may be generated and 

25 stored. The pathologist or cytotechnologist may view the 
mosaic or may also directly view the slide at the 
location of an object of interest in the mosaic for 
further evaluation. The mosaic may be stored on magnetic 
media for future reference or may be transmitted to. a 

30 remote site for review and/or 

storage. The entire process involved in examining a 
single slide takes on the order of 2-15 minutes depending 
on scan area size and the number of detected candidate 
objects of interest. 

35 The present invention has utility in the field of 

oncology for the early detection of minimal residual 
disease ( "micrometastases") . Other useful applications 
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include prenatal diagnosis of fetal cells in maternal 
blood and in the field of infectious diseases to identify 
pathogens and viral loads, alkaline phosphatase 
assessments, reticulocyte counting, and others. 
5 The processing of images acquired in the automated 

scanning of the present invention preferably includes the 
steps of transforming the image to a different color 
space; filtering the transformed image with a low pass 
filter; dynamically thresholding the pixels of the 

10 filtered image to suppress background material ,- 

performing a morphological function to remove artifacts 
from the thresholded image; analyzing the thresholded 
image to determine the presence of one or more regions of 
connected pixels having the same or similar color; and 

15 categorizing every region having a size greater than a 
minimum size as a candidate object of interest. 

According to another aspect of the invention, the 
scan area is automatically determined by scanning the 
slide; acquiring an image at each slide position; 

20 analyzing texture information of each image to detect the 
edges of the specimen; and storing the locations 
corresponding to the detected edges to define the scan 
area .. According to yet another aspect of the invention, 
automated focusing of the optical system is achieved by 

25 initially determining a focal plane from an array of 

points or locations in the scan area. The derived focal 
plane enables subsequent rapid automatic focusing in the 
low power scanning operation. The focal plane is 
determined by determining proper focal positions across 

30 an array of locations and performing an analysis such as 
a least squares fit of the array of focal positions to 
yield a focal plane across the array. Preferably, a focal 
position at each location is determined by incrementing 
the position of a Z stage for a fixed number of coarse 

35 and fine iterations. At each iteration, an image is 

acquired and a pixel variance or other optical parameter 
about a pixel mean for the acquired image is calculated 
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to form a set of variance data. A least squares fit is 
performed on the variance data according to a known 
function. The peak value of the least squares fit curve 
is selected as an estimate of the best focal position. 
5 In another aspect of the present invention, another 

focal position method for high magnification locates a 
region of interest centered about a candidate object of 
interest within a slide which was located during an 
analysis of the low magnification images. The region of 

10 interest is preferably n columns wide, where n is a power 
of 2 . The pixels of this region are then processed using 
a Fast Fourier Transform to generate a spectra of 
component frequencies and corresponding complex magnitude 
for each frequency component . Magnitude of the frequency 

is components which range from 25% to 75% of the maximum 

frequency component are squared and summed to obtain the 
total power for the region of interest. This process is 
repeated for other Z positions and the Z position 
corresponding to the maximum total power for the region 

20 of interest is selected as the best focal position. This 
process is preferably Used to select a Z position for 
regions of interest for slides containing preferably, the 
complex 

neutrophils stained with Fast Red to identify alkaline 
25 phosphatase in cell cytoplasm and counterstained with 
hematoxylin to identify the nucleus of the neutrophil 
cell. This focal method may be used with other stains and 
types of biological specimens, as well. 

According to still another aspect of the invention, 
3 0 a method and apparatus for automated slide handling is 
provided. A slide is mounted onto a slide carrier with a 
number of other slides side-by-side. The slide carrier is 
positioned in an input feeder with other slide carriers 
to 

35 facilitate automatic analysis of a batch of slides. The 
slide carrier is loaded onto the X-Y stage of the optical 
system for the analysis of the slides thereon. 
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Subsequently, the first slide carrier is unloaded into an 
output feeder after automatic image analysis and the next 
carrier is 

automatically loaded, 
s Also provided is an apparatus for processing slides 

according to the methods above . The apparatus includes a 
computer having at least one system processor with image 
processing capability, a computer monitor, an input 
device, a power supply and a microscope subsystem. The 
10 microscope subsystem includes an optical sensing array 

for acquiring images. A two dimensional motion stage for 
sample movement and for focus adjustment, and input and 
output mechanisms for multiple sample analysis and 
storage. 

15 

Brief Description of the Drawings 

The above and other features of the invention including 
various novel details of construction and combinations of 

20 parts will now be more particularly described with 

reference to the accompanying drawings and pointed out in 
the claims. It will be understood that the particular 
apparatus embodying the invention is shown by way of 
illustration only and not as a limitation of the 

25 invention. The principles and features of this invention 
may be employed in varied and numerous embodiments 
without departing from the scope of the invention. 

Fig. 1 is a perspective view of an apparatus for 
30 automated cell analysis embodying the present invention. 

Fig. 2 is a block diagram of the apparatus shown in Fig. 
1. 

35 Fig. 3 is a block diagram of the system processor of Fig. 
2 . 
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Fig. 4 is a plan view of the apparatus of Fig. 1 having 
the housing removed. 

5 Fig. 5 is a side view of a microscope subsystem of the 
apparatus of Fig. 1. 

6a is a top view of a slide carrier for use with the 
apparatus of Fig. 1. 
10 6b is a bottom view of the^ slide carrier of Fig. 6a. 

7a is a top view of an automated slide handling subsystem 
of the apparatus of Fig . 1 . 

is Fig. 7b is a partial cross -sectional view of the 

automated slide handling subsystem of Fig. 7a taken on 
line A-A. 

Fig. 8 is an end view of the input module of the 
20 automated slide handling subsystem. 8a- 8d illustrate the 
input operation of the automatic slide handling 
subsystem. 

9a-9d illustrate the output operation of the automated 
25 slide handling subsystem. 

10 is a flow diagram of the procedure for automatically 
determining a scan area. 

30 11 shows the scan path on a prepared slide in the 
procedure of Fig. 10. 

12 illustrates an image of a field acquired in the 
procedure of Fig. 10. 

35 

13A is a flow diagram of a preferred procedure for 
determining a focal position. 

8 
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Fig. 13B is a flow diagram of a preferred procedure for 
determining a focal position for neutrophils stained with 
Fast Red and counterstained with hematoxylin. 

Fig. 14 is a flow diagram of a procedure for 
automatically determining initial focus. 



Fig. 15 shows an array of slide positions for use in the 
10 procedure of Fig. 14. 

Fig. 16 is a flow diagram of a procedure for automatic 
focusing at a high magnification. 

Fig. 17A is a flow diagram of an overview of the 
is preferred process to locate and identify objects of 
interest in a stained biological specimen on a slide. 

Fig. 17B is a flow diagram of a procedure for color space 
conversion . 

20 

Fig. 18 is a flow diagram of a procedure for background 
suppression via dynamic thresholding. 

Fig. 19 is a flow diagram of a procedure for 
25 morphological processing. 

Fig. 2 0 is a flow diagram of a procedure for blob 
analysis . 



3 0 Fig. 21 is a flow diagram of a procedure for image 
processing at a high magnification. 

Fig. 22 illustrates a mosaic of cell images produced by 
the apparatus . 

35 

Fig. 23 is a flow diagram of a procedure for estimating 
the number of nucleated cells in a field. 
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Fig. 24 illustrates the apparatus functions available in 
a user interface of the apparatus . 

5 DETAILED DESCRIPTION OF THE INVENTION 

Referring now to the figures, an apparatus for 
automated cell analysis of biological specimens is 
generally indicated by reference numeral 10 as shown in 
perspective view in Fig. 1 and in block diagram form in 

10 Fig. 2. The apparatus 10 comprises a microscope subsystem 
32 housed in a housing 12. The housing 12 includes a 
slide carrier input hopper 16 and a slide carrier output 
hopper 18. A door 14 in the housing 12 secures the 
microscope subsystem from the external environment. A 

is computer subsystem comprises a computer 22 having at 

least one system processor 23, and a communications modem 
29. The computer subsystem further includes a computer 
monitor 26 and other external peripherals including 
storage device 21, a pointing device, such as a track 

20 ball device 30, a user input device, such as a touch 

screen, keyboard, or voice recognition unit 28 and color 
printer 35. An external power supply 24 is also shown for 
power outage protection. The apparatus 10 further 
includes an optical sensing array 42, such as a camera, 

25 preferably a CCD camera, for acquiring images. Microscope 
movements are under the control of system processor 23 
through a number of microscope -subsystem functions 
described further in detail. An automatic slide feed 
mechanism in conjunction with X-Y stage 38 provide 

30 automatic slide handling in the apparatus 10 . An 

illumination light source 48 projects light onto the X-Y 
stage 38 which is subsequently imaged through the 
microscope subsystem 32 and acquired through optical 
sensing array 42 for processing by the system processor 

35 23 . A Z stage or focus stage 46 under control of the 
system processor 23 provides displacement of the 

10 
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microscope subsystem in the Z plane for focusing. The 
microscope subsystem 32 further includes a motorized 
objective turret 44 for selection of objectives. 
The purpose of the apparatus 10 is for the 
5 unattended automatic scanning of prepared microscope 
slides for the detection and counting of candidate 
objects of interest such as normal and abnormal cells, 
e.g., tumor cells. The preferred embodiment may be 
utilized for rare event detection in which there may be 

10 only one candidate object of interest per several hundred 
thousand normal cells, e.g., one to five candidate 
objects of interest per 2 square centimeter area 
of the slide. The apparatus 10 automatically locates and 
counts candidate objects of interest and estimates normal 

is cells present in a biological specimen on the basis of 

color, size and shape characteristics. A number of stains 
are used to preferentially stain candidate objects of 
interest and normal cells different colors so that such 
cells can be distinguished from each other. 

20 As noted in the background of the invention, a 

biological specimen may be prepared with a reagent to 
obtain a colored insoluble precipitate. The apparatus of 
the present invention is used to detect this precipitate 
as a candidate object of interest. During operation of 

25 the apparatus 10, a pathologist or laboratory technician 
mounts prepared slides onto slide carriers. A slide 
carrier 60 is illustrated in Fig. 8 and will be described 
further below. Each slide carrier holds up to 4 slides. 
Up to 25 slide carriers are then loaded into input hopper 

30 16. The operator can specify the size, shape and location 
of the area to be scanned or alternatively, the system 
can automatically locate this area. The operator then 
commands the system to begin automated scanning of the 
slides through a graphical user interface. Unattended 

35 scanning begins with the automatic loading of the first 
carrier and slide onto the precision motorized X-Y stage 
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38. A bar code label affixed to the slide or slide 
carrier is read by a bar code reader 3 3 during this 
loading operation. Each slide is then scanned at a user 
selected low microscope magnification, for example, lOx, 
5 to identify candidate cells based on their color, size 
and shape characteristics. The locations, coordinate or 
address of candidate objects of interest are stored, such 
as by X-Y, r-9, polar, vector or other coordinate systems 
well known in the art, until scanning is completed. 

io After the low magnification scanning is completed, 

the apparatus automatically returns to each candidate 
cell, reimages and refocuses at a higher magnification 
such as 4 Ox and performs further analysis to confirm the 
cell candidate. Alternatively, the system may process 

15 low magnification images by reconstructing the image from 
individual fields of view and then determining objects' of 
interest. In this manner, objects of interest that 
overlap more than one objective field of view may be 
identified. The apparatus stores an image of the object 

20 of interest for later review by a pathologist. All 

results and images can be stored to a storage device 21 
such as a removable hard drive, DAT tape, local hard 
drive, optical disk, or transmitted to a remote site for 
review or storage. The stored images for each slide can 

25 be viewed in a mosaic of images for further review. 

Having described the overall operation of the 
apparatus 10 from a high level, the further details of 
the apparatus will now be described. Referring to Fig. 3, 
the microscope subsystem control is shown in more detail . 

30 The microscope control includes a number of subsystems. 
The apparatus system processor 23 controls these 
subsystems . The system 

processor 23 controls a set of motor - control subsystems 
114 through 124 which control the input and output 
35 feeder, the motorized turret 44, the X-Y stage 38, and 

the Z stage 46 (Fig. 2) . The system processor 23 further 
controls an illumination controller 10 6 for control' of 
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substage illumination 48. The light output from the 
halogen light bulb which supplies illumination for the 
system can vary over time due to bulb aging, changes in 
optical alignment, and other factors. In addition, slides 
5 which have been "over stained" can reduce the camera 

exposure to an unacceptable level . In order to compensate 
for these effects, the illumination controller 106 is 
included. This controller is used in conjunction with 
camera and image collection adjustments to compensate for 

10 the variations in light level. The light control software 
samples the output from the camera at intervals {such as 
between loading of slide carriers) , and commands the 
controller to adjust the light or image collection 
functions to the desired levels. In this way, light 

15 control is automatic and transparent to the user and adds 
no additional time to system operation. The system 
processor 2 3 is preferably a high performance processor 
of at least 20 0 MHz, for example the system processor may 
comprise dual parallel Intel Pentium 200 MHZ devices. 

20 Referring now to Figs. 4 and 5, further detail of 

the apparatus 10 is shown. Fig. 4 shows a plan view of 
the apparatus 10 with the housing 12 removed. A portion 
of the automatic slide feed mechanism 37 is shown to the 
left of the microscope subsystem 32 and includes slide 

25 carrier unloading assembly 34 and unloading platform 3 6 
which in conjunction with slide carrier output hopper 18 
function to receive slide carriers which have been ' 
analyzed. Vibration isolation mounts 40, shown in further 
detail in Fig. 5, are provided to isolate the microscope 

30 subsystem 32 from mechanical shock and vibration that can 
occur in a typical laboratory environment. In addition to 
external sources of vibration, the high speed operation 
of the X-Y stage 38 can induce vibration into the 
microscope subsystem 32. Such sources of vibration can be 

35 isolated from the electro-optical subsystems to avoid any 
undesirable effects on image quality. The isolation 
mounts 40 comprise a spring 40a and piston 40b submerged 
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in a high viscosity silicon gel which is enclosed in an 
elastomer membrane bonded to a casing to achieve damping 
factors on the order of 17 to 20%. 

The automatic slide handling feature of the present 
5 invention will now be described. The automated slide 

handling subsystem operates on a single slide carrier at 
a time. A slide carrier 60 is shown in Figs. 6a and 6b 
which provide a top view and a bottom view respectively. 
The slide carrier 60 includes up to four slides 70. The 

10 carrier 60 includes ears 64 for hanging the carrier in 
the output hopper 18. An undercut 66 and pitch rack 68 
are formed at the top edge of the slide carrier 60 for 
mechanical handling of the slide carrier. A keyway cutout 
65 is formed in one side of the carrier 60 to facilitate 

is carrier alignment. A prepared slide 72 mounted on the 
slide carrier 60 includes a sample area 72a and a bar 
code label area 72b. 

Fig. 7a provides a top view of the slide handling 

20 subsystem which comprises a slide input module 15, a 

slide output module 17 and X-Y stage drive belt 50. Fig. 
7b provides a partial cross-sectional view taken along 
line A -A of Fig. 7a. The slide input module 15 comprises 
a slide carrier input hopper 16, loading platform 52 and 

25 slide carrier loading subassembly 54. The input hopper 16 
receives a series of slide carriers 60 (Figs. 6a and 6b) 
in a stack on loading platform 52. A guide key 57 
protrudes from a side of the input hopper 16 to which the 
keyway cutout 65 (Fig. 6a) of the carrier is fit to 

30 achieve proper alignment. The input module 15 further 
includes a revolving indexing cam 56 and a switch 90 
mounted in the loading platform 52, the operation of 
which is described further below. The carrier loading 
subassembly 54 comprises an infeed drive belt 59 driven 

35 by a motor 86. The infeed drive belt 59 includes a pusher 
tab 58 for pushing the slide carrier horizontally toward 
the X-Y stage 38 when the belt is driven. A homing switch 
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95 senses the pusher tab 58 during a revolution of the 
belt 59. Referring specifically to Fig. 7a, the X-Y 
stage 3 8 is shown with x position and y position motors 

96 and 97 respectively which are controlled by the system 
5 processor 23 (Fig. 3) and are not considered part of the 

slide handling subsystem. The X-Y stage 38 further 
includes an aperture 55 for allowing illumination to 
reach the slide carrier. A switch 91 is mounted adjacent 
the aperture 55 for sensing contact with the carrier and 

io thereupon activating a motor 87 to drive stage drive belt 
50 (Fig. 7b) . The drive belt 50 is a double sided timing 
belt having teeth for engaging pitch rack 68 of the 
carrier 60 (Fig. 6b) . 

The slide output module 17 includes slide carrier 

is output hopper 18, unloading platform 3 6 and slide carrier 
unloading subassembly 34. The unloading subassembly 34 
comprises a motor 89 for rotating the unloading platform 
3 6 about shaft 98 during an unloading operation described 
further below. An outfeed gear 93 driven by motor 88 

20 rotatably engages the pitch rack 68 of the carrier 60 
(Fig. 6b) to transport the carrier to a rest position 
against switch 92. A springloaded hold-down mechanism 
holds the carrier in place on the unloading platform 36. 
The slide handling operation will now be described. 

25 Referring to Fig. 8, a series of slide carriers 60 are 
shown stacked in input hopper 16 with the top edges 60a 
aligned. As the slide handling operation begins, the 
indexing cam 56 driven by motor 85 advances one 
revolution to allow only one slide carrier to drop to the 

30 bottom of the hopper 16 and onto the loading platform 52. 

Figs. 8a-8d show the cam action in more detail. The 
cam 56 includes a hub 56a to which are mounted upper and 
lower leaves 56b and 56c respectively. The leaves 56b and 
56c are 

35 semicircular projections oppositely positioned and spaced 
apart vertically. In a first position shown in Fig. 8a, 
the upper leaf 56b supports the bottom carrier at the 
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undercut portion 66. At a position of the cam 56 rotated 
180°, shown in Fig. 8b, the upper leaf 56b no longer 
supports the 

carrier and instead the carrier has dropped slightly and 
5 is supported by the lower leaf 56c. Fig. 8c shows the 
position of the cam 56 rotated 270° wherein the upper 
leaf 56b has rotated sufficiently to begin to engage the 
undercut 66 of the next slide carrier while the opposite 
facing lower leaf 56c still supports the bottom carrier. 

10 After a full rotation of 360° as shown in Fig. 8d, the 

lower leaf 56c has rotated opposite the carrier stack and 
no longer supports the bottom carrier which now rests on 
the loading platform 52. At the same position, the upper 
leaf 56b supports the next carrier for repeating the 

15 cycle. 

Referring again to Figs. 7a and 7b, when the carrier 
drops to the loading platform 52, the contact closes 
switch 90 which activates motors 86 and 87. Motor 86 
drives the infeed drive belt 59 until the pusher tab 58 

20 makes contact with the carrier and pushes the carrier 

onto the X-Y stage drive belt 50. The stage drive belt 50 
advances the carrier until contact is made with switch 
91, the closing of which begins the slide scanning 
process described further herein. 

25 Upon completion of the scanning process, the X-Y stage 3 8 
moves to an unload position and motors 87 and 88 are 
activated to transport the carrier to the unloading 
platform 36 using stage drive belt 50. As noted, motor 88 
drives outfeed gear 93 to engage the carrier pitch rack 

30 68 of the carrier 60 (Fig. 6b) until switch 92 is 

contacted. Closing switch 92 activates motor 89 to rotate 
the unloading platform 36. 

The unloading operation is shown in more detail in 
end views of the output module 17 (Figs. 9a-9d) . In Fig. 

35 9a, the unloading platform 36 is shown in a horizontal 
position supporting a slide carrier 60. The hold-down 
mechanism 94 secures the carrier 60 at one end. Fig. 9b 
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shows the output module 17 after motor 89 has rotated the 
unloading platform 36 to a vertical position, at which 
point the spring loaded hold-down mechanism 94 releases 
the slide carrier 60 into the output hopper 18. The 
5 carrier 60 is supported in the output hopper 18 by means 
of ears 64 {Figs. 6a and 6b) . Fig. 9c shows the unloading 
platform 3 6 being rotated back towards the 20 horizontal 
position. As the platform 3 6 rotates upward, it contacts 
the deposited carrier 60 and the upward movement pushes 

10 the carrier toward the front of the output hopper 18. 

Fig. 9d shows the unloading platform 36 at its original 
horizontal position after having output a series of slide 
carriers 60 to the output hopper 18. 

Having described the overall system and the 

15 automated slide handling feature, the aspects of the 
apparatus 10 relating to scanning, focusing and image 
processing will now be described in further detail. 

In some cases, an operator will know ahead of time 
where the scan area of interest is on the slide. 

20 Conventional preparation of slides for examination 

provides repeatable and known placement of the sample on 
the slide. The operator can therefore instruct the system 
to always scan the same area at the same location of 
every slide which is prepared in this fashion. But there 

2 5 are other times in which the area of interest is not 

known, for example, where slides are prepared manually 
with a known smear technique. One feature of the 
invention automatically determines the scan area using a 
texture or density analysis process. Fig. 10 is a flow 

3 0 diagram that describes the processing associated with the 

automatic location of a scan area. As shown in this 
figure, the basic method is to pre-scan the entire slide 
area to determine texture features that indicate the 
presence of a smear and to discriminate these areas from 
35 dirt and other artifacts. 

At each location of this raster scan, an image such 
as in Fig. 12 is acquired and analyzed for texture 
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information at steps 204 and 206. Since it is desired to 
locate the edges of the smear sample within a given 
image, texture analyses are conducted over areas called 
windows 78, which are smaller than the entire image as 
5 shown in Fig. 12. The process iterates the scan across 
the slide at steps 208,210,212 and 214. 

In the interest of speed, the texture analysis 
process is performed at a lower magnification, preferably 
at a 4x objective. One reason to operate at low 

10 magnification is to image the largest slide area at any 
one time. Since cells do not yet need to be resolved at 
this stage of the overall image analysis, the 4x 
magnification is preferred. On a typical slide, as shown 
in Fig. 11, a portion 72b of the end of the slide 72 is 

is reserved for labeling with identification information. 

Excepting this label area, the entire slide is scanned in 
a raster scan fashion to yield a number of adjacent 
images . Texture values for each window include the pixel 
variance over a window, the difference between the 

20 largest and smallest pixel value within a window, and 
other indicators. The presence of a smear raises the 
texture values compared with a blank area. 

One problem with a smear from the standpoint of 
determining its location is its non-uniform thickness and 

25 texture. For example, the smear is likely to be 

relatively thin at the edges and thicker towards the 
middle due to the nature of the smearing process. To 
accommodate this non-uniformity, texture analysis 
provides a texture value for each analyzed area. The 

30 texture value tends to gradually rise as the scan 

proceeds across a smear from a thin area to a thick area, 
reaches a peak, and then falls off again to a lower value 
as a thin area at the edge is reached. The problem is 
then to decide from the series of texture values the 

3 5 beginning and ending, or the edges, of the smear. The 
texture values are fit to a square wave waveform since 
the texture data does not have sharp beginnings and 
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endings . 

After conducting this scanning and texture 
evaluation operation, one must determine which areas of 
elevated texture values represent the desired smear 74, 
and which represent undesired artifacts. This 'is 
accomplished by fitting a step function, on a line by 
line basis, to the texture values in step 216. This 
function, which resembles a single square wave beginning 
at one edge and ending at the other edge and having an 
amplitude, provides the means for discrimination. The 
amplitude of the best-fit step function is utilized to 
determine whether smear or dirt is present since 
relatively high values indicate smear. If it is decided 
that smear is present, the beginning and ending 
coordinates of this pattern are noted until all lines 
have been processed, and the smear sample area defined at 
218 . 

After an initial focusing operation described 
further herein, the scan area of interest is scanned to 
acquire images for image analysis. The preferred method 
of operation is to initially perform a complete scan of 
the slide at low magnification to identify and locate 
candidate objects of interest, followed by further image 
analysis of the candidate objects of interest at high 
magnification in order to confirm the objects as cells. 
An alternate method of operation is to perform high 
magnification image analysis of each candidate object of 
interest immediately after the object has been identified 
at low magnification. The low magnification scanning then 
resumes, searching for additional candidate objects of 
interest. Since it takes on the order of a few seconds to 
change objectives, this alternate method of operation 
would take longer to complete. 

To identify structure in tissue that cannot be 
captured in a single field of view image or a single 
staining technique, the invention provides a method for 
histological reconstruction to analyze many fields of 
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view on potentially many slides simultaneously. The 
method couples composite images in an automated manner 
for processing and analysis. A slide on which is mounted 
a cellular specimen stained to identify objects of 
5 interest is supported on a motorized stage. An image of 
the cellular specimen is generated, digitized, and stored 
in memory. As the viewing field of the objective lens is 
smaller than the entire cellular specimen, a histological 
reconstruction is made. These stored images of the 

10 entire tissue section may then be placed together in an 
order such that the H/E stained slide is paired with the 
immunohistochemistry slide so that analysis of the images 
may be performed simultaneously. 

The operator can pre -select a magnification level to 

is be used for the scanning operation. A low magnification 
such as the lOx objective is preferred for the scanning 
operation since a larger area can be initially analyzed 
for each acquired scan image. The overall detection 
process for a cell includes a combination of decisions 

20 made at both low (lOx) and high magnification (40x) 

levels. Decision making at the lOx magnification level is 
broader in scope, i.e., objects that loosely fit the 
relevant color, size and shape characteristics are 
identified at the lOx level. 

25 Analysis at the 40x magnification level then 

proceeds to refine the decision making and confirm 
objects as likely cells or candidate objects of interest. 
For example, at the 4 Ox level it is not uncommon to find 
that some objects that were identified at lOx are 

3 0 artifacts which the analysis process will then reject. In 
addition, closely packed objects of interest appearing at 
lOx are separated at the 4 Ox level. 
In a situation where a cell straddles or overlaps 
adjacent image fields, image analysis of the individual 

35 adjacent image fields could result in the cell being 

rejected or undetected. To avoid missing such cells, the 
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scanning operation compensates by overlapping adjacent 
image fields in both the x and y directions. An overlap 
amount greater than half the diameter of an average cell 
is preferred. In the preferred embodiment, the overlap is 
5 specified as a percentage of 

the image field in the x and y directions. Alternatively, 
a reconstruction method as described above may be used to 
reconstruct the image from multiple fields of view. The 
reconstructed image is then analyzed and processed to 

10 find objects of interest. 

The time to complete an image analysis can vary 
depending upon the size of the scan area and the number 
of candidate cells, or objects of interest identified. 
For one example, in the preferred embodiment, a complete 

is image analysis of a scan area of two square centimeters 
in 

which 50 objects of interest are confirmed can be 
performed in about 12 to 15 minutes. This example 
includes not only focusing, scanning and image analysis 

20 but also the saving of 4 Ox images as a mosaic on hard 
drive 21 (Fig. 2) . Consider the utility of the present 
invention in a "rare event" application where there may 
be one, two or a very small number of cells of interest 
located somewhere on the slide. To illustrate the nature 

25 of the problem by analogy, if one were to scale a slide 
to the size of a football field, a tumor cell, for 
example, would be about the size of a bottle cap. The 
problem is then to rapidly search the football field and 
find the very small number of bottle caps and have a 

3 0 high certainty that none have been missed. 

However the scan area is defined, an initial 
focusing operation must be performed on each slide prior 
to scanning. This is required since slides differ, in 
general, in their placement in a carrier. These 

35 differences include slight (but significant) variations 
of tilt of the slide in its carrier. Since each slide 
must remain in focus during scanning, the degree of tilt 
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of each slide must be determined. This is accomplished 
with an initial focusing operation that determines the 
exact degree of tilt, so that focus can be maintained 
automatically during scanning. 
5 The initial focusing operation and other focusing 

operations to be described later utilize a focusing 
method based on processing of images acquired by the 
system. This method was chosen for its simplicity over 
other methods including use of IR beams reflected from 
10 the slide 

surface and use of mechanical gauges. These other methods 
also would not function properly when the specimen is 
protected with a coverslip. The preferred method results 
in lower system cost and improved reliability since no 

15 additional parts need be included to perform focusing. 
Fig. 13A provides a flow diagram describing the "focus 
point" procedure. The basic method relies on the fact 
that the pixel value variance (or standard deviation) 
taken about the pixel value mean is maximum at best 

20 focus. A "brute-force" method could simply step through 
focus, using the computer controlled Z, or focus stage, 
calculate the pixel variance at each step, and return to 
the focus position providing the maximum variance. Such a 
method would be too time consuming. Therefore, additional 

25 features were added as shown in Fig. 13A. 

These features include the determination of pixel 
variance at a relatively coarse number of focal 
positions, and then the fitting of a curve to the data to 
provide a faster means of determining optimal focus. This 

30 basic process is applied in two steps, coarse and fine. 

During the coarse step at 220-230, the Z stage is 
stepped over a user- specif ied range of focus positions, 
with step sizes that are also user-specified. It has been 
found that for coarse focusing, these data are a close 

35 fit to a Gaussian function. Therefore, this initial set 
of variance versus focus position data are least- squares 
fit to a Gaussian function at 228. The location of the 
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peak of this Gaussian curve determines the initial or 
coarse estimate of focus position for input to step 232. 

Following this, a second stepping operation 232-242 
is performed utilizing smaller steps over a smaller focus 
5 range centered on the coarse focus position. Experience 
indicates that data taken over this smaller range are 
generally best fit by a second order polynomial. Once 
this least squares fit is performed at 240, the peak of 
the second order curve provides the fine focus position 
10 at 244. 

Fig. 14 illustrates a procedure for how this 
focusing method is utilized to determine the orientation 
of a slide in its carrier. As shown, focus positions are 
determined, as described above, for a 3 x 3 grid of 

is points centered on the scan area at 264. Should one or 
more of these points lie outside the scan area, the 
method senses this at 266 by virtue of low values of 
pixel variance. In this case, additional points are 
selected closer to the center of the scan area. Fig. 15 

20 shows the initial array of points 80 and new point 82 
selected closer to the center. Once this 
array of focus positions is determined at 268, a least 
squares plane is fit to this data at 270. Focus points 
lying too far above or below this best-fit plane are 

25 discarded at 272 (such as can occur from a dirty cover 
glass over the scan area), and the data is then refit. 
This plane at 274 then provides the desired Z position 
information for maintaining focus during scanning. 

After determination of the best -fit focus plane, the 

30 scan area is scanned in an X raster scan over the scan 
area as described earlier. During scanning, the X stage 
is positioned to the starting point of the scan area, the 
focus (Z) stage is positioned to the best fit focus 
plane, an image is acquired and processed as described 

35 later, and this process is repeated for all points over 
the scan area. In this way, focus is maintained 
automatically without the need for time-consuming 
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refocusing at points during scanning. 

Prior to confirmation of cell objects at a 4 Ox or S.Ox 
level, a refocusing operation is conducted since the use 
of this higher magnification requires more precise focus 
5 than the best- fit plane provides. Fig. 16 provides the 
flow diagram for this process. As may be seen, this 
process is similar to the fine focus method described 
earlier in that the object is to maximize the image pixel 
variance. This is accomplished by stepping through a 

10 range of focus positions with the Z 

stage at 276, 278, calculating the image variance at each 
position at 278, fitting a second order polynomial to 
these data at 282, and calculating the peak of this curve 
to yield an estimate of the best focus position at 284, 

is 286. This final focusing step differs from previous ones 
in that the 

fOcus range and focus step sizes are smaller since this 
magnification requires focus settings to within 0.5 
micron or better. It should be noted that for some 

20 combinations of cell staining characteristics, improved 
focus can be obtained by numerically selecting the focus 
position that provides the largest variance, as opposed 
to selecting the peak of the polynomial. In such cases, 
the polynomial is used to provide an estimate of best 

25 focus, and a final step selects the actual Z position 
giving highest pixel variance. It should also be noted 
that if at any time during the focusing process at 4 Ox or 
60x the parameters indicate that the focus position is 
inadequate, the system automatically reverts to a coarse 

30 focusing process as described above with reference to 
Fig. 13A. This ensures that variations in specimen 
thickness can be accommodated in an expeditious manner. 
For some biological specimens and stains, the focusing 
methods discussed above do not provide optimal focused 

35 results. For example, certain white blood cells known as 
neutrophils may be stained with Fast Red, a commonly 
known stain, to identify alkaline phosphatase in the 
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cytoplasm of the cells. To further identify these cells 
and the material within them, the specimen may be 
count erstained with hematoxylin to identify the nucleus 
of the cells. In cells so treated, the cytoplasm bearing 
5 alkaline phosphatase becomes a shade of red proportionate 
to the amount of alkaline phosphatase in the cytoplasm 
and the nucleus becomes blue. However, where the 
cytoplasm and nucleus overlap, the cell appears purple. 
These color combinations appear to preclude the finding 

10 of a focused Z position using the focus processes 
discussed above. 

In an effort to find a best focal position at high 
magnification, a focus method, such as the one shown in 
Fig. 13B, may be used. That method begins by selecting a 

15 pixel near the center of a candidate object of interest 
(Block 24S) and defining a region of interest centered 
about the selected pixel (Block 250) . Preferably, the 
width of the region of interest is a number of columns 
which is a power of 2. This width preference arises from 

20 subsequent processing of the region of interest 

preferably using a one dimensional Fast Fourier Transform 
(FFT) technique. As is well known within the art, 
processing columns of pixel values using the FFT 
technique is facilitated by making the number of columns 

25 to be processed a power of two. While the height of the 
region of interest is also a power of two in the 
preferred embodiment, it need not be unless a two 
dimensional FFT technique is used to process the region 
of interest. 

30 After the region of interest is selected, the columns of 
pixel values are processed using the preferred one 
dimensional FFT to determine a spectra of frequency 
components for the region of interest (Block 252} . The 
frequency spectra ranges from DC to some highest 

3 5 frequency 

component. For each frequency component, a complex 
magnitude is computed. Preferably, the complex magnitudes 
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for the frequency components which range from 
approximately 25% of the highest component to 
approximately 75% of the highest component are squared 
and summed to determine the total power for the region of 
5 interest (Block 254) . Alternatively, the region of 

interest may be processed with a smoothing window, such 
as a Hanning window, to reduce the spurious high 
frequency components generated by the FFT processing of 
the pixel values in the region of interest . Such 

10 preprocessing of the region of interest permits all 

complex magnitude over the complete frequency range to be 
squared and summed. After the power for a region has been 
computed and stored (Block 256) , a new focal position is 
selected, focus adjusted (Blocks 258, 260), and the 

is process repeated. After each focal position has been 
evaluated, the one having the greatest power factor is 
selected as the one best in focus (Block 262) . 

The following describes the image processing methods 
which are utilized to decide whether a candidate object 

20 of interest such as a stained tumor cell is present in a 
given image, or field, during the scanning process. 
Candidate objects of interest which are detected during 
scanning are reimaged at higher (4 Ox or 60x) 
magnification, the decision confirmed, and a region of 

25 interest for this cell saved for later review. The image 
processing includes color space conversion, low pass 
filtering, background suppression, artifact suppression, 
morphological processing, and blob analysis. One or more 
of these steps can optionally be eliminated. The operator 

30 is provided with an option to configure the system to 

perform any or all of these steps and whether to perform 
certain steps more than once or several times in a row. 
It should also be noted that the sequence of steps may be 
varied 

35 and thereby optimized for specific reagents or reagent 
combinations; however, the sequence described herein is 
preferred. It should be noted that the image processing 
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steps of low pass filtering, thresholding, morphological 
processing, and blob analysis are generally known image 
processing building blocks. 

An overview of the preferred process is shown in 
5 Fig. 17A. The preferred process for identifying and 
locating candidate objects of interest in a stained 
biological specimen on a slide begins with an acquisition 
of images obtained by scanning the slide at low 
magnification (Block 288) . Each image is then converted 

10 from a first color space to a second color space (Block 
2 90) and the color converted image is low pass filtered 
(Block 2 92) . The pixels of the low pass filtered image 
are then compared to a threshold (Block 294) and, 
preferably, those pixels having a value equal to or 

is greater than the threshold are identified as candidate 
object of interest pixels and those less than the 
threshold are determined to be artifact or background 
pixels. The candidate object of interest pixels are then 
morphologically processed to identify groups of candidate 

20 object of interest pixels as candidate objects of 

interest (Block 296) . These candidate objects of interest 
are then compared to blob analysis parameters (Block 298) 
to further differentiate candidate objects of interest 
from objects which do not conform to the blob analysis 

25 parameters and, thus, do not warrant further processing. 
The location of the candidate objects of interest may be 
stored prior to confirmation at high magnification. The 
process continues by determining whether the candidate 
objects of interest have been confirmed (Block 300) . If 

30 they have not been confirmed, the optical system is set 
to high magnification (Block 302) and images of the slide 
at the locations corresponding to the candidate objects 
of interest identified in the low magnification images 
are acquired (Block 288) . These images are then color 

35 converted (Block 290) , low pass filtered (Block 292) , 
compared to a threshold (Block 294) , morphologically 
processed (Block 296) , and compared to blob analysis 
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parameters (Block 298) to confirm which candidate objects 
of interest located from the low magnification images are 
objects of interest. The coordinates of the objects of 
interest are then stored for future reference (Block 
5 303) . 

In general, the candidate objects of interest, such 
as tumor cells, are detected based on a combination of 
characteristics, including size, shape, and color. The 
chain of decision making based on these characteristics 

10 preferably begins with a color space conversion process. 
The optical sensing array coupled to the microscope 
subsystem outputs a color image comprising a matrix of 
64 0 x 48 0 pixels. Each pixel comprises red, green and 
blue (RGB) signal values. 

is It is desirable to transform the matrix of RGB 

values to a different color space because the difference 
between candidate objects of interest and their 
background, such as tumor and normal cells, may be 
determined from their respective colors. Specimens are 

20 generally stained with one or more industry standard 

stains {e.g., DAB, New Fuchsin, AEC) which are "reddish" 
in color. Candidate objects of interest retain more of 
the stain and thus appear red while normal cells remain 
unstained. The specimens may also be counterstained with 

25 hematoxylin so the nuclei of normal cells or cells not 

containing an object of interest appear blue. In addition 
to these objects, dirt and debris can appear as black, 
gray, or can also be lightly stained red or blue 
depending on the staining procedures utilized. The 

30 residual plasma or other fluids also present on a smear 
may also possess some color. 

In the color conversion operation, a ratio of two of 
the RGB signal values is formed to provide a means for 
discriminating color information. With three signal 

35 values for each pixel, nine different ratios can be. 

formed: R/R, R/G, R/B, G/G, G/B, G/R, B/B, B/G, B/R. The 
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optimal ratio to select depends upon the range of color 
information expected in the slide specimen. As noted 
above, typical stains used for detecting candidate 
objects of interest such as tumor cells are predominantly 
5 red, as opposed to predominantly green or blue. Thus, the 
pixels of a cell of interest which has been stained 
contain a red component which is larger than either the 
green or blue components . A ratio of red divided by blue 
(R/B) provides a value which is greater than one for 

10 tumor cells but is approximately one for any clear or 
white areas on the slide. Since the remaining cells, 
i.e., normal cells, typically are stained blue, the R/B 
ratio for pixels of these latter cells yields values of 
less than one. The R/B ratio is preferred for clearly 

is separating the color information typical in these 
applications . 

Fig. 17B illustrates the flow diagram by which this 
conversion is performed. In the interest of processing 
speed, the conversion is implemented with a look up 

20 table. The use of a look up table for color conversion 
accomplishes three functions: 1) performing a division 
operation; 2) scaling the result for processing as an 
image having pixel values ranging from 0 to 255; and 3) 
defining objects which have low pixel values in each 

25 color band (R,G,B) as "black" to avoid infinite ratios 
{i.e., dividing by zero). These "black" objects are 
typically staining artifacts or can be edges of bubbles 
caused by pasting a coverglass over the specimen. 
Once the look up table is built at 304 for the specific 

30 color ratio (i.e., choices of tumor and nucleated cell 
stains) , each pixel in the original RGB image is 
converted at 308 to produce the output. Since it is of 
interest to separate the red stained tumor cells from 
blue stained normal ones, the ratio of color values is 

35 then scaled by a user specified factor. As an example, 
for a factor of 128 and the ratio of (red pixel 
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value) /(blue pixel value), clear areas on the slide would 
have a ratio of 1 scaled by 128 for a final X value of 
128. Pixels which lie in red stained tumor cells would 
have X value greater than 128, while blue stained nuclei 
5 of normal cells would have value less than 128. In this 
way, the desired objects of interest can be numerically 
discriminated. The resulting 64 0 x 48 0 pixel matrix, 
referred to as the X- image, is a gray scale image having 
values ranging from 0 to 255. 

io Other methods exist for discriminating color 

information. One classical method converts the RGB color 
information into another color space, such as HSI (hue, 
saturation, intensity) space. In such a space, distinctly 
different hues such as red, blue, green, yellow, may be 

15 readily separated. In addition, relatively lightly 

stained objects may be distinguished from more intensely 
stained ones by virtue of differing saturations. However, 
converting from RGB space to HSI space requires more 
complex computation. Conversion to a color ratio is 

20 faster; for example, a full image can be converted by the 
ratio technique of the present invention in about 3 0 ms 
while an HSI conversion can take several seconds. 

In yet another approach, one could obtain color 
information by taking a single color channel from the 

25 optical sensing array. As an example, consider a blue 
channel, in which objects that are red are relatively 
dark. Objects which are blue, or white, are relatively 
light in the blue channel. In principle, one could take a 
single color channel, and simply set a threshold wherein 

3 0 everything darker than some threshold is categorized as a 
candidate object of interest, for example, a tumor cell, 
because it is red and hence dark in the channel being 
reviewed. However, one problem with 

the single channel approach occurs where illumination is 
35 not uniform. Non-uniformity of illumination results in 
non-uniformity across the pixel values in any color 
channel, for example, tending to peak in the middle of 
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the image and dropping off at the edges where the 
illumination falls off. Performing thresholding on this 
non-uniform color information runs into problems, as the 
edges sometimes fall below the threshold, and therefore 
s it becomes more difficult to pick the appropriate 

threshold level. However, with the ratio technique, if 
the values of the red channel fall off from center to 
edge, then the values of the blue channel also fall off 
center to edge, resulting in a uniform ratio at non- 

10 uniform lighting. Thus, the ratio technique is more 
immune to illumination. 

As described, the color conversion scheme is 
relatively insensitive to changes in color balance, i.e., 
the relative outputs of the red, green, and blue 

is channels. However, some control is necessary to avoid 

camera saturation, or inadequate exposures in any one of 
the color bands. This color balancing is performed 
automatically by utilizing a calibration slide consisting 
of a clear area, and a "dark" area having a known optical 

20 transmission or density. The system obtains images from 
the clear and "dark" areas, calculates "white" and 
"black" adjustments for the image-frame grabber or 
digitizer processor 25, and thereby provides correct 
color balance. 

25 In addition to the color balance control, certain 

mechanical alignments are automated in this process. The 
center point in the field of view for the various 
microscope objectives as measured on the slide can vary 
by several Cor several tens of) microns. This is the 

30 result of slight variations in position of the microscope 
objectives 44a as determined by the turret 44 (Fig. 4) , 
small variations in alignment of the objectives with 
respect to the system optical axis, and other factors. 
Since it is desired that each microscope objective be 

35 centered at the same point, these mechanical offsets must 
be measured and automatically compensated. 
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This is accomplished by imaging a test slide which 
contains a recognizable feature or mark. An image of this 
pattern is obtained by the system with a given objective, 
and the position of the mark determined. The system then 
5 rotates the turret to the next lens objective, obtains an 
image of the test object, and its position is 
redetermined- Apparent changes in position of the test 
mark are recorded for this objective. This process is 
continued for all objectives. 

10 Once these spatial offsets have been determined, they are 
automatically compensated for by moving the stage 3 8 by 
an equal (but opposite) amount of offset during changes 
in objective. In this way, as different lens objectives 
are selected, there is no apparent shift in center point 

is or 

area viewed. A low pass filtering process precedes 
thresholding. An objective of thresholding is to obtain a 
pixel image matrix having only candidate objects of 
interest, such as tumor cells above a threshold level and 
20 everything else below it. However, an actual acquired 
image will contain 

noise. The noise can take several forms, including white 
noise and artifacts. The microscope slide can have small 
fragments of debris that pick up color in the staining 
25 process and these are known as artifacts. These artifacts 
are generally small and scattered areas, on the order of 
a few 

pixels, which are above the threshold. The purpose of low 
pass filtering is to essentially blur or smear the entire 
30 color converted image. The low pass filtering process 
will smear artifacts more than larger objects of 
interest, such as tumor cells and thereby eliminate or 
reduce the number of 

artifacts that pass the thresholding process. The result 
3 5 is a cleaner thresholded image downstream. In the low 
pass filter process, a '3 x 3 matrix of coefficients is 
applied to each pixel in the 640 x 480 x- image. A 
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preferred coefficient matrix is as follows: 

1/9 1/9 1/9 
1/9 1/9 1/9 

5 1/9 1/9 1/9 

At each pixel location, a 3 x 3 matrix comprising the 
pixel of interest and its neighbors is multiplied by the 
coefficient matrix and summed to yield a single value for 

10 the pixel of interest. The output of this spatial 

convolution process is again a 64 0 x 480 matrix. As an 
example, consider a case where the center pixel and only 
the center pixel, has a value of 255 and each of its 
other neighbors, top left, top, top right and so forth, 

15 have values of 0 . 

This singular white pixel case corresponds to a 
small ■ object . The result of the matrix multiplication and 
addition using the coefficient matrix is a value of 1/9 
(255) or 28 for the center pixel, a value which is below 

20 the nominal threshold of 128. Now consider another case 
in which all the pixels have a value of 255 corresponding 
to a large object. Performing the low pass filtering 
operation on a 3 x 3 matrix for this case yields a value 
of 255 for the center pixel. Thus, large objects retain 

25 their values while small objects are reduced in amplitude 
or eliminated. In the preferred method of operation, the 
low pass filtering process is performed on the X image 
twice in succession. 

In order to separate objects of interest, such as a 

3 0 tumor cell in the x image from other objects and 
background, a thresholding operation is performed 
designed to set pixels within cells of interest to a 
value of 255, and all other areas to 0. Thresholding 
ideally yields an image in which cells of interest are 

35 white and the remainder of the image is black. A problem 
one faces in thresholding is where to set the threshold 
level. One cannot simply assume that cells of interest 
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are indicated by any pixel value above the nominal 
threshold of 12 8. A typical imaging system may use an 
incandescent halogen light bulb as a light source. As the 
bulb ages, the relative amounts of red and blue output 
5 can change. The tendency as the bulb ages is for the blue 
to drop off more than the red and the green. To 
accommodate for this light source variation over time, a 
dynamic thresholding process is used whereby the 
threshold is adjusted dynamically for each acquired 

10 image. Thus, for each 64 0 x 480 image, a single threshold 
value is derived specific to that image. 
As shown in Fig. 18, the basic method is to calculate, 
for each field, the mean X value, and the standard 
deviation about this mean at 312. The threshold is then 

is set at 314 to the mean plus an amount defined by the 
product of a (user specified) factor and the standard 
deviation of the color converted pixel values. The 
standard deviation correlates to the structure and number 
of objects in the image. Preferably, the user specified 

20 factor is in the range of approximately 1.5. to 2.5. The 
factor is selected to be in the lower end of the range 
for slides in which the stain has primarily remained 
within cell boundaries and the factor is selected to be 
in the upper end of the range for slides in which the 

25 stain is pervasively present throughout the slide. In 
this way, as areas are encountered on the slide with 
greater or lower background intensities, the threshold 
may be raised or lowered to help reduce background 
objects. With this method, the threshold changes in step 

30 with the aging of the light source such that the effects 
of the aging are canceled 

out. The image matrix resulting at 316 from the 
thresholding step is a binary image of black (0) and 
white (255) pixels. 
35 As is often the case with thresholding operations such as 
that described above, some undesired areas will lie above 
the threshold value due to noise, small stained cell 
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fragments, and other artifacts. It is desired and 
possible to eliminate these artifacts by virtue of their 
small size compared with legitimate cells of interest. 
Morphological processes' are utilized to perform this 
5 function. 

Morphological processing is similar to the low pass 
filter convolution process described earlier except' that 
it is applied to a binary image. Similar to spatial 
convolution, the morphological process traverses an input 

10 image matrix, pixel by pixel, and places the processed 
pixels in an output matrix. Rather than calculating a 
weighted sum of the neighboring pixels as in the low pass 
convolution process, the morphological process uses set 
theory operations to combine neighboring pixels in a 

is nonlinear fashion. 

Erosion is a process whereby a single pixel layer is 
taken away from the edge of an object. Dilation is the 
opposite process which adds a single pixel layer to the 
edges of an object. The power of morphological processing 

20 is that it provides for further discrimination to 
eliminate small objects that have survived the 
thresholding process and yet are not likely tumor cells. 
The erosion and dilation processes that make up a 
morphological "open" operation preferably make small 

25 objects disappear yet allow large objects to remain. 

Morphological processing of binary images is described in 
detail in "Digital Image Processing", pages 127-137, G.A. 
Baxes, John Wiley & Sons, (1994) . 

Fig. 19 illustrates the flow diagram for this 

30 process. A single morphological open consists of a single 
morphological erosion 32 0 followed by a single 
morphological dilation 322. Multiple "opens" consist of 
multiple erosions followed by multiple dilations. In the 
preferred embodiment, one or two morphological opens are 

35 found to be suitable. 

At this point in the processing chain, the processed 
image contains thresholded objects of interest, such as 
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tumor cells (if any were present in the original image) , 
and possibly some residual artifacts that were too large 
to be eliminated by the processes above. 

Fig. 20 provides a flow diagram illustrating a blob 
5 analysis performed to determine the number, size, and 
location of objects in the thresholded image. A blob is 
defined as a region of connected pixels having the same 
"color", in this case, a value of 255. Processing is 
performed over the entire image to determine the number 

10 of such regions at 324 and to determine the area and 

coordinates for each detected blob at 326. Comparison of 
the size of each blob to a known minimum area at 328 for 
a tumor cell allows a refinement in decisions about which 
objects are objects of interest, such as tumor cells, and 

15 which are artifacts. The term ""coordinate" or 

""address ' ' is used to mean a particular location on a 
slide or sample. The coordinate or address can be 
identified by any number of means including, for example, 
X-Y coordinates, r-6 coordinates, and others recognized 

20 by those skilled in the art. The location of objects 

identified as cells of interest in this stage are saved 
for the final 40x reimaging step described below. Objects 
not passing the size test are disregarded as artifacts. 
The processing chain described above identifies 

25 objects at the scanning magnification as cells of 

interest candidates. As illustrated in Fig. 21, at the 
completion of scanning, the system switches to the 4 Ox 
magnification objective at 33 0, and each candidate is 
reimaged to confirm the identification 332. Each 40x 

30 image- is reprocessed at 334 using the same steps as 
described above but with test parameters suitably 
modified for the higher magnification (e.g. area) . At 
336, a region of interest centered on each confirmed cell 
is saved to the hard drive for review by the 

35 pathologist. 

As noted earlier, a mosaic of saved images is made 
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available for viewing by the pathologist. As shown in 
Fig. 22, a series of images of cells which have been 
confirmed by the image analysis is presented in the 
mosaic 150. The pathologist can then visually inspect the 
images to make a determination whether to accept (152) or 
reject (153) each cell image. Such a determination can be 
noted and saved with the mosaic of images for generating 
a printed report . 

In addition to saving the image of the cell and its 
region, the cell coordinates are saved should the 
pathologist wish to directly view the cell through the 
oculars or on the image monitor. In this case, the 
pathologist reloads the slide carrier, selects the slide 
and cell for review from a mosaic of cell images, and the 
system automatically positions the cell under the 
microscope for viewing. 

It has been found that normal cells whose nuclei 
have been stained with hematoxylin are often quite 
numerous, numbering in the thousands per lOx image. Since 
these cells are so numerous, and since they tend to 
clump, counting each individual nucleated cell would add 
an excessive processing burden, at the expense of speed, 
and would not necessarily provide an accurate count due 
to clumping. The apparatus performs an estimation process 
in which the total area of each field that is stained 
hematoxylin blue is measured and this area is divided by 
the average size of a nucleated cell. Fig. 23 outlines 
this process. In this process, a single color band (the 
red channel provides the best contrast for blue stained 
nucleated cells) is processed by calculating the average 
pixel value for each field at 342, establishing two 
threshold values (high and low) as indicated at 344, 346, 
and counting the number of pixels between these two 
values at 348. In the absence of dirt, or other opaque 
debris, this provides a count of the number of 
predominantly blue pixels. By dividing this value by the 
average area for a nucleated cell at 350, and looping 
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over all fields at 352, an approximate cell count is 
obtained. Preliminary testing of this process indicates 
an accuracy with +/- 15%. It should be noted that for 
some slide preparation techniques, the size of nucleated 
5 cells can be significantly larger than the typical size. 
The operator can select the appropriate nucleated cell 
size to compensate for these characteristics. 

As with any imaging system, there is some loss of 
modulation transfer {i.e. contrast) due to the modulation 

10 transfer function (MTF) characteristics of the imaging 
optics, camera, electronics, and other components. Since 
it is desired to save "high quality" images of cells of 
interest both for pathologist review and for archival 
purposes, it is desired to compensate for these MTF 

15 losses. 

An MTF compensation, or MTFC, is performed as a digital 
process applied to the acquired digital images. A digital 
filter is utilized to restore the high spatial frequency 
content of the images upon storage, while maintaining low 

20 noise levels. With this MTFC technology, image quality is 
enhanced, or restored, through the use of digital 
processing methods as opposed to conventional oil- 
immersion or other hardware based methods. MTFC is 
described further in 

25 "The Image Processing Handbook," pages 225 and 337, J. C. 
Rues, CRC Press (1995) . 

Referring to Fig. 24, the functions available in a 
user interface of the apparatus 10 are shown. From the 
user interface, which is presented graphically on 

30 computer monitor 26, an operator can select among 
apparatus functions which include acquisition 402, 
analysts 404, and system configuration 406. At the 
acquisition level 402, the operator can select between 
manual 4 08 and automatic 410 modes of operation. In the 

3 5 manual mode, the operator is presented with manual 
operations 409. Patient information 414 regarding an 
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assay can be entered at 412. In the analysis level 404, 
review 416 and report 418 functions are made available. 
At the review level 416, the operator can select a 
montage function 420. At this montage level, a 
pathologist can perform diagnostic review functions 
including visiting an image 422, accept/reject of cells 
424, nucleated cell counting 426, accept/reject of cell 
counts 428, and 

saving of pages at 430. The report level 418 allows- an 
operator to generate patient reports 432. In the 
configuration level 406, the operator can select to 
configure preferences at 434, input operator information 
437 at 436, create a system log at 438, and toggle a menu 
panel at 440. The configuration preferences include scan 
area selection functions at 442, 452; montage 
specifications at 444, bar code handling at 446, default 
cell counting at 448, stain selection at 450, and scan 
objective selection at 454. 
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CLAIMS 



1. An apparatus for automatic image analysis of a slide 
5 having a biological specimen, comprising: 

a computer comprising 

at least one system processor; 
a monitor in operable communication with the computer; 
an input device in communication with the computer; 
10 an optical system in operable communication with the 
computer, comprising 

a movable stage; 

an automated loading and unloading member for 
loading and unloading of the slide; 
15 an identification member for identifying a 

processing of a slide; 

an optical sensing array in optical communication with 
the stage configured to acquire an image at each 
location in a scan area; 
20 an image processor in electrical communication with the 
sensing array and operable to process each 
image to detect candidate objects of interest in the 
image ; 

a storage member for storing the location of a 
25 candidate object of interest; and 

a storage device for storing each image. 

2. The apparatus of claim 1, wherein the system 
processor comprises a high performance processor of at 

30 least 90 MHz . 

3. The apparatus of claim 1, wherein the input device 
is selected from keyboard, trackball, mouse, or touch 
screen monitor. 

35 
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4. The apparatus of claim 1, wherein the identification 
member detects a bar code. 

5. The apparatus of claim 1, wherein the optical 
sensing array is a CCD array. 

6. The apparatus of claim 1, wherein the image is a 
digital image . 

7. The apparatus of claim 1, wherein the storage device 
is a magnetic disk. 

8. The apparatus of claim 1, wherein the storage device 
is an optical disk. 

9. A method for automatic image analysis of a slide 
having a biological specimen, comprising: 

positioning the slide having a biological specimen 
on a stage which is optically coupled to an optical' 
sensing array; 

acquiring an image of the biological specimen; 

processing the image ; 

identifying a candidate object of interest; 
storing the coordinates of the candidate object of 
interest ; 

acquiring a higher magnification image at the 
coordinates of the object of interest; 

processing the higher magnification image; and 
storing the higher magnification image. 

10. The method of claim 9, wherein pixels of the image 
are transformed from a first color space to a second 
color space to differentiate candidate object of interest 
pixels from background pixels. 
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11. The method of claim 10, wherein the first color 
space includes red, green, and blue components for each 
pixel and the transforming step includes forming a ratio 
between two components of the red, blue and green 

5 components for each pixel in the first color space to 
transform the pixels to the second color space. 

12. The method of claim 11, further comprising selecting 
a gray scale value for each pixel in the second color 

10 space which corresponds to the ratio of components in the 
first color space. 

13. The method of claim 10, wherein the first color 
space includes red, green, and blue components for each 

15 pixel and the transforming step includes converting 

components of the red, blue and green components for each 
pixel in the first color space to pixel values in a hue, 
saturation, and intensity space. 

20 14. The method of claim 10, wherein the first color 

space includes red, green, and blue components for each 
pixel and the transforming step includes comparing pixel 
values for a single component for each pixel to a 
threshold to identify pixels having a components value 

25 equal to or greater than said threshold as candidate 

object of interest pixels and pixels having a component 
value less than the threshold as background pixels. 

15. The method of claim 9, further comprising 

30 reconstructing the sample from a plurality of images. 

16. The method of claim 9 or 15, further comprising, 
morphologically processing candidate object of interest 
pixels to identify artifact pixels and identifying the 

35 candidate object of interest from the remaining candidate 
object of interest pixels not identified as artifact 
pixels. 
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17. The method of claim 16, further comprising filtering 
said candidate object of interest pixels with a low pass 
filter prior to morphologically processing the low pass 
filtered candidate object of interest pixels. 

5 

18. The method of claim 17, further comprising comparing 
said low pass filtered candidate object of interest 
pixels to a threshold prior to morphologically processing 
the candidate object of interest pixels which have values 

10 greater than or equal to the threshold value. 

19. The method of claim 18, further comprising 

computing a mean value of said candidate object 
of interest pixels; 
is specifying a threshold factor; 

computing a standard deviation for the 
candidate object of interest pixels,- and 

setting the threshold to the sum of the mean 
value and the product of the threshold factor and the 
20 standard deviation prior to comparing the candidate 
object of interest pixels to the threshold. 

20. The method of claim 16, further comprising: 

grouping said morphologically processed 
25 candidate object of interest pixels into regions of 

connected candidate object of interest pixels to identify 
objects of interest; 

comparing said objects of interest to blob 
analysis parameters; and 
30 storing location coordinates of the candidate 

objects of interest having an area corresponding to the 
blob analysis parameters . 



35 



21. The method of claim 20, wherein the method is 
performed on images acquired at a low magnification and 
the method further comprises: 
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adjusting an optical system viewing the slide 
from which the objets of interest were identified to a 
higher magnification; 

acquiring a higher magnification image of the 
5 slide at the corresponding location coordinates for each 
candidate object of interest ; 

transforming pixels of the higher magnification 
image in the fist color space to a second color space to 
differentiation higher magnification candidate objects of 
10 interest pixels from background pixels; and 

identifying higher magnification objets of 
interest form the candidate object of interest pixels in 
the second color space. 

is 22. The method of claim 21, further comprising 

morphologically precessing the higher magnification' 
candidate object of interest pixels to identify artifact 
pixels and identifying the higher magnification objects 
of interest form the remaining higher magnification 

20 candidate object of interest pixels not identified as 
artifact pixels. 

23. The method of claim 22, further comprising filtering 
said higher magnification candidate object of interest 

25 pixels with a low pass filter prior to morphologically 
processing the low pass filtered higher magnification 
candidate object of interest pixels. 

24. The method of claim 23, further comprising, 

30 comparing said low pass filtered higher magnification 

candidate object of interest pixels to a threshold prior 
to morphologically processing the higher magnification 
candidate object of interest pixels which have values 
greater than or equal to the threshold value. 
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25. The method of claim 24, further comprising: 
computing a mean value of said higher 

magnification candidate object of interest pixels ,- 
specifying a threshold factor; 
5 computing a standard deviation for the higher 

magnification candidate object of interest pixels; and 
setting the threshold to the sum of the man 

value and the product of the threshold factor and the 

standard deviation prior to comparing the higher 
10 magnification candidate object of interest pixels to the 

threshold. 



26. The method of claim 14, further comprising: 
grouping said low pass filtered higher 
is magnification candidate object of interest pixels into 
regions of connected higher magnification candidate 
object of interest pixels to identify higher 
magnification object of interest; 

comparing said higher magnification objects of 
20 interest to blob analysis parameters; and 

storing the location coordinates of the higher 
magnification objects of interest corresponding to the 
blob analysis parameters. 



25 27. The method of claim 21, wherein the optical system 
is initially focused prior to performing the low 
magnification processing. 



28. The method of claim 27, wherein the initial focusing 
30 of the optical system further comprises: 

a. ) positioning the optical system at an 
initial Z stage position; 

b. ) acquiring at low magnification an image of 
a slide having a stained biological specimen thereon and 

35 calculating a pixel variance about a pixel mean for the 
acquired image; 

c. ) incrementing the position of the Z stage; 
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d. ) repeating steps (b) and (c) fora fixed 
number of course iterations to form a fist set of 
variance data; 

e. ) performing a least squares fit of the 
5 first set of variance data to a first function; 

f. ) position the Z stage at a position near 
the peak of the first iteration; 

g. ) repeating steps (b) and (c) for a fixed 
number of fine iterations to form a second set of 

10 variance data; 

h. ) performing a least squared fit of the 
second set of variance data to a second function; 

i . ) selecting the peak value of the least 
squares fit curve as an estimate of the best focal 

15 position; and 

j . ) performing the above steps for an array of 
stage positions to form an array of focal positions and 
performing a least squares fit of the array of focal 
positions to yield a least squares for focal plane. 

20 

29. The method of claim 27, wherein the initial focusing 
of the optical system further comprises the steps of: 

a.) positioning the optical system at an 
initial Z stage position; 
25 b.) acquiring an image and calculating a pixel 

variance about a pixel mean for the acquired image; 

c. ) incrementing the position of the Z stage; 

d. ) repeating steps (b) and (c) or a fixed 
number of iterations ; 

30 e.) performing a least squares fit of the 

variance data to a known function; and 

f.) selecting the peak value of the least 
squares fit curve as an estimate of the best focal 
position. 
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30. The method of claim 21, wherein adjusting the 
optical system further comprises the steps of: 

a.) positioning the optical system at an 
initial Z stage position; 
5 b.) acquiring an image and selecting a center 

pixel of a candidate object of interest; 

c. ) defining a region of interest centered 
about the selected center pixel; 

d. ) performing a fast fourier transform of 

10 said region of interest to identify frequency components 
for the region of interest and complex magnitudes for the 
frequency components ; 

e. ) computing a power value by summing the 
square of the complex magnitudes for the frequency 

is components that are within the rage of frequencies of 25% 
to 75% of a maximum frequency component of the fast 
fourier transform of the region of interest; 

f. ) incrementing the position of the Z stage; 

g. ) repeating steps (b) to (e) for a fixed 
20 number of iterations; and 

h. ) selecting the Z stage position 
corresponding to the largest power value as the best 
focal position. 

25 31. The method of claim 21, wherein adjusting the 
optical system further comprises the steps of: 

a. ) positioning the optical system at an 
initial Z stage position; 

b. ) acquiring an image and selecting a center 
30 pixel of a candidate object of interest; 

c . ) defining a region of interest centered 
about the selected center pixel; 

d. ) applying a Hanning window function to the 
region of interest ,- 

35 e.) performing a fast fourier transform of 

said region of interest following the application of the 
Hanning window function to identify frequency components 
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of the region of interest and complex magnitudes for the 
frequency components; 

f. ) computing a power value by summing the 
square of the complex magnitudes for the frequency 

5 components for the fast fourier transform of the region 
of interest; 

g. ) incrementing the position of the Z stage; 

h. ) repeating steps (b) to (e) for a fixed 
number of iterations; and 

10 i.) selecting the Z stage position 

corresponding to he largest power value as the best focal 
position. 
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